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(57) Abstract: Apparatus and method for presenting a highly spatially accurate visualisation of a scene from which measurements 
can be taken. A sensor is located in relation to a camera, and provides positional characteristics of the camera as it collects frames of 
video images. Using the positional characteristics the frames are corrected. The corrected frames are then synchronised to form an 
accurate mosaic of a scene. Example embodiments are described where the moving camera is used to survey or inspect underwater 
^ apparatus, roads, runways, railways, crime or accident scenes, archaeological digs and the inside of boilers, chimneys and pipelines. 
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I IMi^GING AND MEASUREMENT SYSTEM 

2 

3 The present invention relates to video mosaicing and, in 

4 particular, to a method and system for providing a highly 

5 spatially accurate visualisation of a scene from which 

6 measurements can be taken. 
7 

8 A video mosaic is a composite image produced by stitching 

9 together frames from a video sequence such that similar 

10 regions overlap. The. output gives a representation of 

11 the scene as a whole, rather than a sequential view of 

12 parts of that scene, as in the case of a video survey of 

13 an area. One of the best known applications of this 

14 technique being the creation of panoramic photographs of 

15 a scene. 
16 

17 In publishing and image retouching applications the 

18 mosaics are manually generated which is a costly and time 

19 consuming process. More recently a system for 

20 automatically generating a mosaic has been suggested, US 

21 Patent .5, 649, 032, which provides the possibility for 

22 real-time video mosaicing. This Patent details^ 

23 applications for display of an image, compression of an 

24 image for storage and when constructed, to a surveillance 



CONFIRMATION COPY 
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1 system suitable for determining enemy movement on a 

2 battlefield, a burglar entering a warehouse, and the 

3 like. 
4 

5 Video mosaics constructed in this fashion are not suited 

6 to applications involving the making of accurate 

7 measurements for the following reasons. 
8 

9 Firstly, it is vital to perform a camera calibration 

10 procedure to estimate and hence correct for the 

11 distortions caused by the internal geometry of the 

12 camera- Uncorrected, these distortions will significantly 

13 degrade the accuracy of any measurements made from the 

14 mosaic. 
15 

16 Secondly, the nature of the accumulation of errors in the 

17 estimation of rotations between frames leads a drift 

18 characteristic of a ^^random walk'' which will seriously 

19 degrade the accuracy of long range measurements . 
20 

21 Finally, non-translational changes in the camera position 

22 (e.g. pitch and roll) will lead to perspective changes 

23 between frames which will also degrade the positional 

24 accuracy of the constructed mosaic. Although it is 

25 possible to estimate the variation in camera attitude 

26 from the video frames, the accumulation of the associated 

27 . errors would again lead to degradation in measurement 

28 accuracy. 
29 

30 It is an object of the present invention to provide a 

31 measurement system and method using video mosaicing which 

32 obviates or mitigates at least some of the disadvantages 

33 in the prior art. 
34 
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It is further object of at least one embodiment of the 
present invention to provide a measurement system and 
method to provide a highly spatially accurate 
visualisation of a scene from which measurements can be 
taken - 

It is a still further object of at least one embodiment 
of the present invention to provide a measurement system 
and method from which one can make measurements of a 
scene to millimetre resolution. 

According to a first aspect of the present invention 
there is provided apparatus for presenting a highly 
spatially accurate visualisation of a scene from which 
measurements can be taken, the apparatus comprising: 

at least one camera for recording a plurality of 
frames of video images of the scene; 

at least one sensor mounted in relation to the 
camera for recording sensor data on positional 
characteristics of the camera as the at least one 



image processing means including a first module for 
synchronising the frames with the sensor data to 
form corrected frames; and a second module for 
constructing an accurate mosaic from the corrected 
frames • 



camera is moved with respect to the scene; and 



By first correcting the video frames prior to the 
mosaiced image being formed, distortions present in the 
frames recorded by the one or more cameras can be removed 
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1 and so enhance the spatial resolution over the entire 

2 mosaiced image. 

3 ■ 

4 Preferably the at least one camera is a video camera 

5 capturing 2 dimensional digital images. 
6 

7 The at least one sensor may comprise any sensor capable 

8 of making a positional measurement. Preferably the at 

9 least one sensor comprise sensors making a measurement 

10 relating to attitude or distance. Preferably also the at 

11 least one sensor comprises a digital compass. 

12 Advantageously the digital ccanpass records roll, pitch 

13 and yaw. Preferably also, the at least one sensor 

14 comprises an altimeter and/or bathymetric sensor. . 



16 Advantageously the camera (s) and sensor (s) are mounted on 

17 a moving platform. In use the platform may be mounted on 

18 a vehicle to allow movement of the camera (s) and 



21 The apparatus may further include a calibration system 

22 from which the at least one camera is calibrated. In this 

23 way spherical lens distortion e.g. pincushion distortion 

24 and barrel distortion can be corrected prior to use of 

25 the camera (s). Further non-equal scaling of the pixels in 

26 the X and y axis is corrected together with a skew of the 

27 two image axis from the perpendicular. 
28 

29 Advantageously the calibration system includes a 

30 chessboard pattern or regular grid. This provides for 

31 multiple images to be taken from multiple viewpoints so 

32 that the distortions can be estimated and compensated 

33 for . 



15 



19 



sensor (s) over or through the scene to be imaged. 



20 



34 
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1 Preferably the first module performs a perspective 

2 correction to the images using the sensor data. 

3 Preferably also^ the corrected frames are of a 

4 preselected position with reference to .the scene. 

5 Optionally the corrected frames may be of preselected 

6 attitude and distance. 
7 

8 Preferably the second module accomplishes video mosaicing 

9 via a correlation technique based on frequency contents 
10 of the images being compared. 

11 

12 Preferably the apparatus further includes display means 

13 for providing a visual image of the mosaic. Preferably 

14 also the apparatus further comprises data storage means 

15 to allow the mosaic to be stored for viewing at a later 

16 time- 
17 

18 Preferably also the apparatus includes a graphic user 

19 interface (GUI) , More preferably the GUI is included with 

20 the display system. Advantageously the GUI includes means 

21 to allow a user to select and make measurements between 

22 points in the visual image of the mosaic. Optionally the 

23 GUI provides a user with means to control the movement of 

24 the at least one camera. 
25 

26 According to a second aspect of the present invention 

27 there is provided a method for presenting a highly 

28 spatially accurate' visualisation of a scene from which 

29 measurements can be taken, the method comprising the 

30 steps; 
31 

32 (a) recording a plurality of frames of video images 

33 of the scene from a camera; 
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1 (b) recording sensor data on positional 

2 characteristics of the camera as the camera is 

3 moved with respect to the scene; 

4 (c) synchronising the frames with the sensor data 

5 to form corrected frames; and 

6 (d) constructing an accua«ate mosaic from the 

7 corrected frames. 
8 

9 Preferably the method includes the step of calibrating 

10 the camera prior to step (a) , This calibration may remove 

11 distortion effects within the camera, 
12 

13 Preferably the step of calibrating includes the step of 

14 taking multiple images of a chessboard pattern or regular 

15 grid from multiple viewpoints and further estimating and 

16 compensating for the distortions. 
17 

18 Preferably the synchronisation step includes the step of 

19 performing a perspective correction to the images using 

20 the sensor data. 
21 

22 Preferably also the step' of video mosaicing is achieved 

23 using a correlation technique based on frequency contents 

24 of the images being compared, 
25 

26 Preferably the method further includes the step of 

27 providing a visual image of the mosaic. 
28 

29 Advantageously the method further includes the step of 

30 taking a measurement from the visual image. 
31 

32 Optionally the method may include t the step of storing the 

33 images so that they may be accessed by spatial position. 
34 
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1 This method may advantageously be used to record crime 

2 scenes, accident scenes, archaeological digs and the like 

3 where traditional methods of image recordal and distance 

4 measurement are time consuming. Additionally by storing 

5 the mosaiced images, distances previously not measured 

6 within the scene can be regenerated and accurately 

7 measured without having to reconstruct or preserve the 

8 original scene. 
9 

10 According to a third aspect of the present invention 

11 there is provided a method of performing a survey in a 

12 fluid, the method comprising the steps of; 
13 

14 (a) mounting a camera and a plurality of sensors on a 

15 platform capable of movement in the fluid; 

16 (b) moving the platform through the fluid while 

17 recording visual images on the camera and taking 

18 sensor data relating to the attitude and distance 

19 of the platform from objects of interest within 

20 the fluid; 

21 (c) synchronising the visual images to the sensor data 

22 to provide corrected visual images relating to a 

23 fixed distance and d:ttitude; 

24 (d) video mosaicing the images to form an accurate 

25 video mosaic as a visual image of the scene 

26 surveyed. 
27 

28 Preferably the method includes the step of precalibrating 

29 the camera to compensate for distorting artefacts 

30 inherent within the camera. 
31 

32 Preferably the method includes the step of displaying the 

33 visual image. More preferably the method includes the 

34 step of taking a measurement from the visual image. 
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Preferably the fluid is water, so that measurements can 
be made underwater. In this way pipe spool dimensions 
can be taken underwater as can determination be made of 
the degree of damage or degradation of pipelines. 

Advantageously the platform may be mounted on an 
autonomous underwater vehicle (AUV) or a remotely 
operated vehicle (ROV) . Alternatively the platform may 
be mounted on a PIG (pipeline "inspection gauge) , so that 
the camera can be moved through a pipeline to inspect the 
inner surface of the pipeline. 

Preferably the method includes the step of storing the 
mosaiced images for viewing later. 

Embodiments of the present invention will now be 
described, by way of example only, with reference to the 
following Figures, of which: 

Figure 1 is a schematic diagram of a first 
embodiment of. the present invention; 

Figure 2 is a schematic diagram of a second 
embodiment of the present inventions- 
Figure 3 is a flow diagram depicting the stages of 
the sensor data integration with the algorithms 
required for the construction of the measurement 
mosaic of the second embodiment; 

Figure 4 depicts a schematic of the camera pose 
alteration required to correct for perspective in 
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each of the image frairfes by application of the pitch 
and roll sensor data in the second embodiment; 

Figure 5 shows a flow diagram of the method applied 
when correcting images for the sensor roll and pitch 
data concurrently with the camera calibration 
correction as in the second embodiment; 

Figure 6 is a schematic diagram of a third 
embodiment of the present invention; and 

Figure 7 is a schematic • diagram of a fourth 
embodiment of the present -invention. 

Referring initially to Figure 1 there is shown imaging 
apparatus, generally indicated by reference numeral 10, 
according to a first embodiment of the present invention. 
Apparatus 10 comprises a camera 12 mounted with sensors 
14,16, The camera 12 captures a series of frames of video 
images as the camera 12 and sensors 14,16 are moved over 
an object 18. During this movement the sensors 14,16 
record data on the attitude and distance of the camera 12 
from the object 18. The sensor data and video images are 
input an image processor, generally indicated at 20. The 
processor 20 includes a first module 22 in which the 
frames are synchronised with the sensor data, as will be 



described hereinafter. The first module 22 outputs 
corrected video image from which is constructed a video 
mosaic in the second module 24, as described hereinafter. 
The video mosaic of the object 18 is displayed on a 
monitor 2 6 of a personal computer. Using a graphical user 
interface 28 of the personal computer a user can select 
points on the video mosaic and obtain distance 
measurements of the object 18. The measurements provide 
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millimetre accuracy over 20 metre distances to the 
object. This is achieved by correcting variations in 
pixel dimensions with the sensor data and/or camera 



calibration^ described hereinafter, and using the sensor 
data to also provide a deteirmination of pixel dimensions 
in terms of real metric units, 

Figure 2 depicts a schematic diagram of a second 
embodiment of the present invention illustrating the 
hardware and the high level processes. This embodiment 
consists of an instrumented camera platform, generally 
indicated by reference numeral 30, incorporating a video 
camera 32 which may be analogue or digital, a digital 
compass 34 and an altimeter sensor 36. The sensors 34,36 
measure the attitude (roll, pitch and yaw/heading) of the 
platform 30 and the distance from the camera platfoimi 30 



to an object being viewed... In underwater applications, 
an additional bathymetric sensor may be used to measure 
the depth of submergence of the camera platform 30. Thus 
the platform 30 will be mounted on a suitable vehicle 35 
e.g. underwater remotely operated vehicle (ROV) , aircraft 
or even a hand-held mounting and moved across the scene 
of interest. As in the first embodiment, the video and 
sensor data is made available to the operator 37 of the 
system for live display. Additionally, the video and 
sensor data is stored 38 in a format which allows precise 
synchronization between the video and sensor data. The 
stored data 38 may be retrieved and used to construct a 
video mosaic image 40 representing a plan view of the 
scene being surveyed whei;e pixel scale is maintained 
throughout the image. During the construction of this 
mosaic image corrections are applied to the video frames 
to correct the inherent distortions due to the video 
camera and to compensate for the effects of camera 



wo 2004/029878 



PCT/GB2003/004163 




11 



1 

2 
3 
4 
5 
6 
7 
8 
9 
10 
11 
12 
13 
14 
15 
16 
17 
18 
19 
20 
21 
22 
23 
24 
25 
26 
27 
28 
29 
30 
31 
32 
33 
34 



platform attitude and distance to the viewed scene. 
These corrections ensure that the constructed mosaic 
image 40 is an accurate representation of the scene being 
surveyed, with the relative scales and positions of the 
objects contained within the scene being preserved as 
well as possible. Once constructed^ it is possible to 
obtain measurements 42 of -objects contained within the 
mosaic image using a graphical user interface. 

Figure 3 depicts a flow diagram of the stages required to 
construct the video mosaic image. The first stage in 
this process is to acquire a frame of video data 50 and 
the corresponding sensor data 52 for this frame, from the 
storage unit 38. The video frame 50 is then corrected to 
compensate for the effects of the camera distortion and 
the camera platform attitude 54 . This stage requires 
knowledge of the camera internal parameters which are 
estimated by a calibration method described later, and 
the pitch and roll angles 56 recorded by the digital 



compass 34. The correcte4 image 58 is then input into 
the mosaicing procedure 60 where it is compared with the 
previous corrected video frame 50 in the video sequence. 
This procedure attempts to estimate the translation in x 
and y axes between the two frames by comparing the 
correlations between the frames in the frequency domain. 
The rotation between frames and the scale change between 
frames is determined from the compass heading and 
altitude/depth information 62. The next stage 64 is to 
apply the transformation parameters to the new frame and 
incorporate it into the final mosaic image 66, a process 
known as ^'stitching". Finally the pixel size may be 
determined by the 'use of a calibration target placed in 
the scene, or directly ^from the camera calibration 
parameters and altimeter sensor data. 
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We shall consider the steps taken in the method in more 
detail. Beginning with the camera 32, all cameras suffer 
from various forms of distortion. This distortion arises 
from certain artefacts inherent to the internal camera 
geometric and optical characteristics (otherwise known as 
the intrinsic parameters) . These artefacts include: 

(a) spherical 'lens distortion about the principal 



for this type of distortion are pincushion 
distortion and barrel distortion; 

(b) non-equal scaling of pixels in the x and y-axis. 
This is arrived at through the estimation of the 
effective camera focal length in both the x and y 
pixel scales; and 

(c) a skew of the two image axes from the 
perpendicular . 

For high accuracy* mosaicing the parameters leading to 
these distortions must be estimat^ed and compensated for. 
In order to correctly estimate these parameters images 
taken from multiple viewpoints of a regular grid;, or 
chessboard type pattern are used. The corner positions 
are located in each image using a corner detection 
algorithm. The resulting points are then used as input 
to a camera calibration algorithm as well documented in 
the literature. 

The estimated intrinsic parameter matrix A is of the form 



point of the system. The*" two common definitions 
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0 1_ 

2 

3 where a and J3 are the focal lengths in x and y pixels 

4 respectively,. ;^ is a factor accounting for skew due to 

5 non-rectangular pixels, and (Wq^'^o) the principle point 

6 (that is the perpendicular projection of the camera focal 

7 point onto the image plane) • 
8 

9 During the creation of the mosaic, the integration of the 

10 sensor data is performed in two phases; as is illustrated 

11 in Figure 4. The first of these involves the use of the 

12 pitch and roll measurements 56 from the compass 34 to 

13 perform a perspective correction on each of the frames 

14 prior the mosaicing procedure 60. A diagram showing the 

15 situation modelled by this correction is provided in 

16 figure 4. When correcting for perspective the new camera 

17 position 70 is at the same height 72 as the original 

18 viewpoint 74, not the slant range distance 7 6a,b,c. Thus 

19 any correction for perturbations in pitch or roll will 

20 not be misinterpreted as a change in camera height, which 

21 may be considered either as a separate process handled 

22 within the mosaicijig procedure 60 itself, or gained from 

23 the bathymetric sensor readings'. * 
24 

25 This perspective correction 54 is performed concurrently 

26 with the camera calibration correction 55 following the 

27 steps outlined in Figure 5. Figure 5 illustrates the 

28 steps applied to all pixel positions in the corrected 

29 image 58. Starting with the corrected image pixel 

30 position 58, we obtain the corresponding pixel position 

31 in the cameras true reference frame 82, we then obtain 
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1 the position in captured image distorted by the camera 

2 calibration parameters 84, interpolate for value at 

3 resulting subpixel level 86 and insert interpolate value 

4 into initial corrected image pixel position 88. 
5 

6 Concatenating these two operations in this way saves on 

7 both processing time and memory requirements. These 

8 processes combine mathematically in the following way: 
9 

10 If u is the corrected pixel position , the corresponding 

11 position in the reference frame of the camera, normalised 

12 according the camera focal length in y pixels and 

13 centred on the principle point (Wq^Vq), is 

14 c'=[(c/\c2",C3'')/c4*'-(Wo,Vo)]/>ff where c''= PR^R^P^'u . The pitch 

15 and roll are represented by the rotation matrices Rj^ and 

16 Ry respectively, with P being the perspective projection 

17 matrix which maps real world coordinates onto image 

18 coordinates. Following this the pixel position in the 

19 captured image is calculated as £= At^.c* . The scalar r^.. 

20 represents the radial distortion applied at the camera 

21 reference frame coordinate & . The matrix A is as 

22 defined previously. 
23 

24 In estimating interframe mosaicing parameters of video 

25 sequences there are currently two types of method 

26 available. The first uses feature matching within the 

27 image to locate objects and then to align the two frames 

28 based on the positions of common objects. The second 

29 method is frequency based, and uses the properties of the 

30 Fourier transform. 
31 
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1 Given the volume of data involved (a typical capture rate 

2 being 25 frames per second) it is important that we 

3 utilise a technique which will provide a fast data 

4 throughput^ whilst also being highly accurate in a 

5 multitude of working environments. In order to achieve 

6 these goals ^ the preferred embodiment employs the 

7 correlation technique based on the frequency content of 

8 the images being compared. This approach has two main 

9 advantages; firstly, regions which would appear 

10 relatively featureless, that is those not containing 

11 strong corners, linear features, and such like, still 

12 contain a wealth of frequency information representative 

13 of the scene. This is extremely important when mosaicing 

14 regions of the seabed for example, as definite features 

15 (such as corners or edges) may be sparsely distributed; 

16 if indeed they exist at all; and secondly, the fact that 

17 this technique is based on the Fourier transform means 

18 that it opens itself immediately to fast implementation 

19 through highly optimized software and hardware solutions, 
20 

21 The second phase of integration is applied in tandem with 

22 the frequency correlation technique and incorporates both 

23 the altimeter and heading readings, 
24 

25 The mosaicing technique- is capable of estimating the 

26 rotations between adjacent frames in the mosaic to an 

27 extremely high degree of accuracy. Unfortunately, the 

28 nature of the accumulation of the errors corresponds to a 

29 stochastic process called a ^^random walk". This has the 

30 effect of leading to a drift in the estimated track. For 

31 short range mosaics this effect is limited and may be 

32 discounted, thus ' allowing use of Fourier rotation 

33 measurements. However, for long *range mosaics this will 

34 not be the case. In order to overcome this, the yaw data 
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1 is utilised from the digital compass to provide a stable 

2 reference for the camera heading- This greatly increases 

3 the overall accuracy of the reconstructed mosaic. 
4 

5 For each image comparison, the interfrarae rotation and 

6 scaling values are obtained from the difference in the 

7 heading and bathymetric readings for that image pair. 

8 The second image is then corrected to the same 

9 orientation and scale of the first. This way only the 

10 translation in x and y pixels need be estimated. Having 

11 obtained the necessary parameters of the differences in 

12 position of the two images, they can be placed in their 

13 correct relative positions. The next frame is then 

14 analysed in a similar manner and added to the evolving 

15 mosaic image. 
16 

17 We shall now give a description of the implementation 

18 procedures used in this invention for translation 

19 estimation in Fourier space. 
20 

21 In Fourier space, translation is a phase shift. We 

22 therefore must utilise the differences in the phase to 

23 determine the translational shift! Let the two images be 

24 described by and fji^^y) where (x,y) represents a 

25 pixel at this position. Then for a translation (dx^cfy) the 

26 two frames are related by 
27 

28 /a (x, y) = f^(x + clx,y + dy) 

29 

30 The Fourier transform magnitudes of these two images are 

31 the same since the translation only affects the phases. 

32 Let our original images be of size (cols^rows) , then each of 

33 these axes represents a range of 2;r radians. So a shift 
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of dx pixels corresponds to 27C.dxlcols shift in phase for 
the coliamn axis. Similarly, a shift of dy pixels 
corresponds to iTV^dyl rows shift in phase for the row axis. 

To determine a translation, we Fourier transform the 
original images, compute the magnitude {M) and phases 
(^) of each of the pixels and subtract the phases of each 
pixel to get d4> . We then take the average of the 
magnitudes (they should be the same) and the phase 
differences and compute a new set of real (91) and 
imaginary (3) valxaes as 5R = Mcos(<i^) and 5 = Msin(c/^). These 
(91,3) values are then inverse *Fourier transformed to 
produce an image- Ideally, this image will have a single 
bright pixel at a position (x^y) , which represents the 
translation between the original two images, whereupon a 
subpixel translation estimation may be made. 

It is not always that case that the peak is unique 
however. When we have translation close to zero, the 
gained true peak is often distorted by a secondary peak 
at the origin- For this reason we place a lower 
acceptance bound on the translation. If the gained 
translation is lower that this, then the current new 
frame is discarded-, and the next is compared to the same 
initial frame. This process * has the added speed 

advantage that frames are only stitched into the mosaic 
if a reasonable translation has occurred. 

A final point to note concerning this technique is that 
we must first window the intensity values to be Fourier 
transformed, ensuring that they are reduced to zero at 
the boundary. This removes the step discontinuities at 
the boundaries, making the periodic image, implied when 
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stepping into the Fourier domain, appear continuous in 
all directions. 

Following acquisition of the interframe mosaicing 
parameters it remains for the video images to be stitched 
into a single mosaic so that measurements between imaged 
positions may be achieved. This is performed using a 
similar philosophy to that adopted when correcting for 
perspective and camera calibration. Given a pixel 

position within the mosaic, what was the corresponding 
sub-pixel position in the original frame? The 
construction of the mosaic is also performed in such a 
way as to minimise the amount of memory required to 
contain the result. 

In order to determine this mapping we first generate the 
camera track file containing the frame centre positions, 
orientations, and scale factors from the parameter file 
output by the mosaicing algorithm. This is done through 
accumulation of local translations, rotations, and 
scaling factors, each having undergone a rotation and 
scaling to make them local to the mosaic reference frame. 

Following this, we may calculate the coordinates of the 
i^^ frame pixel position (Xyj^j^^j), in terms of the 
corresponding mosaic pixel position (x„,,j/,„), as 
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where 0^ and z, are the rotation and scaling values which 
place the f"' frame into the mosaic, the size of area 
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1 required to fully contain the frame in the mosaic is 

2 p^^xp^^ pixels, and the original frame size is fc^fr 

3 pixels,* We then interpolate the sub-pixel value at 

4 position C-^/, »>^/;) frame i, and place this value into 

5 mosaic pixel position {x^^y^) . 
6 

7 Given the stitched mosaic it remains to make a 

8 measurement between selected points in the final result, 

9 In order to accomplish this, the pixel size must be 

10 determined through use of either a calibration target 

11 placed in the scene, or through use of the camera 

12 calibration parameters and altimeter sensor data. 

13 Following this calibration, the distance in pixels 

14 between the selected points is multiplied by the true 

15 distance subtended by each pixel to provide an accurate 

16 length measurement. 
17 

18 The apparatus and 'method of the present invention lends 

19 itself to the following applications particularly as 

20 applied to underwater surveying: 
21 

22 (a) Metrology, through the measurement of physical 

23 dimensions in difficult to access environments; 
24 

25 (b) Geo-ref erencing - in conventional video surveys 

26 the data is stored in a video format where each 

27 part of the survey is accessed by frame number. 

28 Under the present invention a survey can be 

29 stored as one or more mosaiced images which can 

30 advantageously be accessed by spatial position 

31 and integrated with other geo-ref erenced data 

32 such as maps, sidescan *sonar, and engineering 

33 drawings; 
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1 

2 (c) Video compression - while video recording of a 

3 survey requires vast storage capacity and leads 

4 to data being stored on difficult to access 

5 magnetic tape media or in compressed forms on a 

6 computer, the present invention provides a 

7 compact data size as redundant information when 

8 images overlap is removed. This is done with very 

9 little degradation to the image quality compared 

10 to video compression methods . It is also possible 

11 to reconstruct a video of the original video 

12 survey; and 
13 

14 (d) Navigation as the video mosaicing process 

15 involves the measurement of translations 

16 rotations and scalings that are present in the 

17 video sequence^ the apparatus can provide 

18 navigational information about the platform on 

19 which it may be mounted. As the navigational 

20 information extracted from the video sequence may 

21 be extremely accurate (<lcm) over short ranges, 

22 the information can be used to aid positioning of 

23 equipment, station Jiolding and offers a potential 

24 benefit to the development of a synthetic 

25 aperture sonar system. 
26 

27 It will be appreciated that the second embodiment could 

28 be adapted to inspect ships' hulls in order to check for 

29 hull integrity or the prevention of smuggling or 

30 terrorist threats. In this application the camera (s) and 

31 sensors are mounted onto a remotely operated vehicle 

32 (ROV) which is used to scan the hull of the ship. In 

33 this configuration, the sensors could include an 

34 altimeter to measure distance between the camera and ship 
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1 hull/ and a digital compass unit to measure the platform 

2 attitude- The sensor data can be used to apply scaling 

3 and perspective corrections respectively to the camera 

4 frames, prior to mosaicing the video frames into a large 

5 image. The mosaic image may be used to identify the 

6 position of any area of interest on the ship's hull. 
7 

8 A further application of this methodology is that of 

9 internal pipe-like structure inspection, where pipe-like 

10 structures include pipelines, boilers, and chimneys for 

11 example. In this embodiment a system 100 includes a 

12 plurality of cameras 90 are placed in a circular 

13 arrangement as shown in figure - 6 *to provide a 360 degree 

14 field of view, and images gathered of the surrounding 

15 surface 92. Lighting sources 94 are placed adjacent to 

16 the cameras 90; suitably illuminating the surface 92 

17 being inspected. The cameras 90 are synchronised with 

18 images gathered instantaneously being distortion 

19 corrected depending on the camera calibration parameters, 

20 arrangement of the cameras^ and position of the camera 

21 system within the pipe structure, thereby providing 

22 images from which the accurate measurements of distances 

23 along the pipe sidewall 92 may be obtained. The position 

24 within the structure can be determined by separate range 

25 finding sensors 95 mounted locally to each camera and 

26 synchronised with that camera, tHese supply the distance 

27 to the pipe structure sidewall of that camera. Via a 

28 processor 98 the instantaneously grabbed images are then 

29 accumulated into a mosaiced image strip containing the 

30 entire imaged surface at that particular moment in time. 

31 The system 100 can be propelled through a boiler or pipe 

32 like structure via any means including gravity (a 

33 vertical pipeline or chimney for example), a pulley 

34 system pulling/pushing the setup, or by attaching to the 
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1 camera rig an arrangement of support struts with wheels, 

2 these may be motoafised or pushed/pulled through the pipe 

3 structure by some external mearfs- As the number of 

4 strips accumulates over time they are automatically 

5 stitched to form a mosaic of the surface under 

6 inspection; the inside of a pipe, chimney, or boiler. 
7 

8 A yet further application of an embodiment of invention 

9 described here is in the inspection of roads, runways and 

10 railway lines. In this embodiment the system 102 could 

11 consist of video cameras 104 mounted on a suitable 

12 vehicle 106 facing towards the ground with the addition 

13 of suitable lighting 108 to illuminate the surface being 

14 inspected. In this configuration the additional sensors 

15 could include a GPS receiver 110 that can be used to 

16 provide additional global pbsitioning information 

17 synchronised to the video data. The video frames will be 

18 corrected for camera and perspective distortion prior to 

19 input to the mosaicing operation in the processor 112. A 

20 video mosaic constructed from the combined (in the case 

21 of more than one camera) and corrected video frames will 

22 be generated. This image may be used to identify and 

23 measure surface defects and to determine global positions 

24 of these defects. The incorporation of GPS positional 

25 information can further enable the generated mosaic image 

26 to be referenced to a geographical information system 

27 (GIS) . 
28 

29 The main advantage of the present invention is that it 

30 provides a video mosaic image from which measurements 

31 with millimetre accuracy can be taken. High spatial 

32 resolution is attainable by fusing the sensor data with 

33 the video images and then reconstructing the mosaic from 

34 a selected reference point. This allows measurements to 



wo 2004/029878 



PCT/GB2003/004163 



23 



1 
2 
3 
4 
5 
6 
7 
8 
9 
10 
11 
12 
13 
14 
15 
16 
17 
18 
19 
20 
21 
22 
23 
24 
25 
26 
27 
28 
29 
30 
31 
32 



be made from the video mosaic as the pixel dimensions are 
provided in terms of metric units scaled from the objects 
being surveyed. Use of a correlation technique based on 
the frequency content of the images being compared 
provides the advaatages of allowing imaging of generally 
featureless scenes such as ' the seabed and as the 
technique is based on the Fourier Transform the data can 
be processed in real time through the implementation of 
highly optimised software* and hardware solutions . 

Further the present invention provides advantages over 
traditional ways of obtaining measurements. Firstly, it 
may be used in environments where it is either hazardous 
or difficult to use conventional manual measurement 
methods . For example the measurement of pipeline spool 
pieces on the seafloor, can be conducted by mounting the 
camera and sensors on an ROV which can be flown over the 
two ends of the pipeline to be connected by the spool 
piece. Currently a method involving triangulation of 
acoustic transceivers is employed for this application. 
This is a time consuming method which requires the use of 
divers and some expert knowledge. A second advantage is 
that in the case of scenes containing a number of objects 
that must have their positions or separations recorded, a 
survey can be conducted and the measurements made at a 
later time, with the minimum of delay incurred at the 
scene. This would be a considerable benefit in recording 
accident scenes or archaeological digs. 

It will be appreciated by those skilled in the art that 
various modifications may be made to the invention herein 
described without departing from the scope thereof. 
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Apparatus for presenting a highly spatially accurate 
visualisation of a scene from which measurements can 
be taken, the apparatus comprising: 

at least one camera for recording a plurality of 
frames of video images of the scene; 

at least one sensor mounted in relation to the 
camera for recording sensor data on positional 
characteristics of the camera as the at least one 
camera is moved with respect to the scene; and 

image processing means including a first module for 
synchronising the frames with the sensor data to 
form corrected frames; and a second module for 
constructing an accurate mosaic from the corrected 
frames . 

Apparatus as claimed in Claim 1 wherein the at least 
one camera is a video camera capturing 2 dimensional 
digital images. 

Apparatus as claimed in Claim 1 or Claim 2 wherein 
the at least one sensor comprises a sensor capable 
of making a positional measurement. 

Apparatus as claimed in Claim 3 wherein the at least 
one sensor comprises a digital compass. 

Apparatus as claimed in Claim 3 or Claim 4 wherein 
the at least one sensor comprises an altimeter 
and/or bathymetric sensor. 
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1 

2 6. Apparatus as claimed in any preceding Claim wherein 

3 the camera (s) and sensor (s) are mounted on a moving 

4 platform. 
5 

6 7. Apparatus as claimed in any preceding Claim wherein 

7 the apparatus further includes a calibration system 

8 from which the at least one camera is calibrated. 
9 

10 8. Apparatus as claimed in any pjreceding Claim wherein 

11 the first module performs a perspective correction 

12 to the images using the sensor data. 
13 

14 9. Apparatus as claimed in any preceding Claim wherein 

15 the second module accomplishes video mosaicing via a 

16 correlation technique based on frequency contents of 

17 the images being compared. 
18 

19 10. Apparatus as claimed in any preceding Claim wherein 

20 the apparatus further includes display means for 

21 providing a visual image of the mosaic . 
22 

23 11. Apparatus as claimed in any f>receding Claim wherein 

24 the apparatus further comprises data storage means 

25 to allow the mosaic to be stored. 
26 

27 12. Apparatus as claimed in any preceding Claim wherein 

28 the apparatus includes a graphic user interface 

29 (GUI). 
30 

31 13. A method for presenting a highly spatially accurate 

32 visualisation of a scene from which measurements can 

33 be taken, the method comprising the steps; 
34 
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(a) recording a plurality of frames of video images 
of the scene from a camera; 

(b) recording sensor data on positional 
characteristics of the camera as the camera is 
moved with respect to the scene; 

(c) synchronising the frames with the sensor data 
to form corrected frames; and 

(d) constructing an accurate mosaic from the 
corrected frames. 

14 . A method as claimed in Claim 13 wherein the method 
includes the step of calibrating the camera prior to 
step (a) . 

15. A method as claimed in Claim 13 or Claim 14 wherein 
the synchronisation step includes the step of 
performing a perspective correction to the images 
using the sensor data. 

16. A method as claimed in any one of Claims 13 to 15 
wherein the step of video mosaicing is achieved 
using a correlation technique based on frequency 
contents of the images being compared. 

17. A method as claimed in any one of Claims 13 to 16 
wherein the method further includes the step of 
providing a visual ima^e of the mosaic. 

18 . A method as claimed in any one of Claims 13 to 17 
wherein the method further includes the step of 
taking a measurement from the visual image - 

19. A method as claimed in any one of Claims 13 to 18 
wherein the method includes the step of storing the 
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images so that they may be accessed by spatial 
position, 

20. A method of performing^ a survey in a fluid, the 
method comprising the steps of; 

(a) mounting a camera and a plurality of sensors on 
a platform capable of movement in the fluid; 

(b) moving the platform through the fluid while 
recording visual images on the camera and 
taking sensor data relating to the attitude and 
distance of the platform from objects of 
interest within the fluid; 

(c) synchronising the visual images to the sensor 
data to provide corrected visual images 
relating to a fixed distance and attitude; 



(d) video mosaicing the images to form an accurate 
video mosaic as a visual image of the scene 
surveyed . 

21. A method as claimed in Claim 20 wherein the method 
includes the step of precalibrating the camera to 
compensate for distorting artefacts inherent within 
the camera . 

22. A method as claimed in Claim 20 or 21 wherein the 
method includes the step of displaying the visual 



23. A method as claimed in, any' one of Claims 20 to 22 
wherein the method includes the step of taking a 
measurement from the visual image. 
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24. A method as claimed in any one of Claims 20 to 23 
wherein the platfomn is mounted on a remotely 
operated vehicle (ROV) • 



25. 



A method as* claimed in any one of Claims 20 to 24 
wherein the method includes the step of storing the 
mosaiced images for viewing later. 
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